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ABSTRACT 

In order to move away from reliance on a s_**gle 
questionnaire in university course evaluation and move towards a more 
formative evaluation, a method was developed for helping a team of 
instructors specify and evaluate their course and program goals. The 
method involves small group workshops in which instructors are asked 
to describe their program and state their objectives, in terms of 
student behavior. Discrimination training on clarity of objectives is 
provided. Written statements are then circulated to all instructors 
to test consensus. An application of the method to 30 instructors in 
a medical school department is discussed. (Author) 
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Intro duction 

If course evaluation in universities is to contribute 
substantially to the impx"ovement of student learning, it 
must move away from reliance on a single questionnaire given 
to ''tudents at one point in time* This type of evaluation 
does not often provide a sufficient amount of detailed and 
timely feedback needed to develop an instructional system 
nor does it provide complete summative information to assess 
courses as complex educational program.^. A more formative 
evaluation is needed if significant changes in course design 
and outcomes can be expecced. 
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Most of the current theories of formative evaluation 
involve the specification of objectives and goals which must 
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themselves be evaluated as an early step. Goal specifica- 
tion and evaluation can be a beginning point for the forinative 
evaluation of a university course or departmental program. 
To fill this need, a method v;as developed for helping a team 
of instructors (e.g., a department) specify and evaluate 
their goals. The method was developed as part of an eval- 
uation of the program offered by a relatively large Depart- 
ment at McGill University. More than 30 instructors were 
involved in an application of the method. The authors served 
as evaluation consultants to the project. 

Attempting to generate and evaluate departmental goals 
v;ith a group offers special problems of reaching consensus. 
Even when one is consulting with an individual instructor, 
departmental goals may need to be clarified before course 
goals can be specified appropriately. This would be particu- 
larly true of courses in a sequence. Previous experience 
of the authors in working with large (N< 5) groups of 
instructors revealed the same problems one encounters in 
cocktail party discussions over art objects or political 
candidates: different philosophical points of viev/, differing 
amounts of prior experience in discussing the proi:)lem at 
hand, and various levels of interesl- in the general topic 
make consensus rather diflxoult if not irapossible, to 
achieve. 
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So it is with getting groups of instructors to agree 
on the collective objectives of their field of expertise • 
Some instructors feel that stating objectives is essential 
to designing and evaluating instruction; others feel that 
the "good" students will manage to discover what they need . 
to know* Some instructors are quite preoccupied with dis- 
cussing educational matters, whereas others feel quite dif- 
ferently. Also, some instructors have had direct experience 
in writing objectives, while others have not. 

Description of Method 

The method, which might best be called "Divide and 
Conquer," is outlined in Figure 1, The method involves a 
series of discussion-workshops which were held with small 



Insert Figure 1 about here 

groups of instructors {3-u) meeting separately and reprej^ent- 
ing different aspects of the departmental program. Small 
groups of this size have been found to encourage discussion 
and reduce some of the negative effects of large group 
discussions, which can be dominated by a few speakers. These 
workshops lasted only one and a half hours each, but were 
designed to be task oriented to prevent rambling discussions. 
The purpose of workshops was to provide informal training 
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on how to specify objectives and to generate a first dj'aft 
of written objectives for the course or program represented 
by the group. Emphasis was on fluency, thr.t is, generating 
as many objectives as possible, and objectivity in stating 
course outcomes* Therefore, instructors were asKed to a) 
describe their sections of the program in terms of student 
and teacher behavior and b) state their objective in terms 
of student behavior. One of the long-range purposes of these 
workshops was to be able to document for future evaluation 
any discrepancies between course descriptions and course 
intent. 

Participants v;ero encouraged to make verbal statements 
of objectives during the workshops. The consultants probed 
for further clarification and attempted to train discriminations 
between poorly-stated and well-stated objectives. Each 
group had a chairman who was responsible for preparing the 
written statement of objectives developed in the workshop* 
He sent a copy of this statement to the consultaaits after the 
v/orkshop for a further check on the form of the objectives 
(e.g., v/hether they referred to actual student behavior, 
whether they v/ere clear) . 

It should be pointed out that chairmen of these groups 
were chosen by the Department chairman and the major criterion 
for these choices was previous experience in "educational 
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matters." Severcil of the group leaders had previous 
experience in writing behavioral objectives and naturally 
this made the process work more efficiently. 

The consultants then rev;rote those objectives which 
were unclear^ asked group leaders for clarification \.hen neces- 
sary^ and compiled a list of objectives generated by all 
of the groups. 

This list v/as then circulated to all members of the 
department and each one was asked to cominent on their relevance^ 
clarity^ and redundancy by making editorial com^ients m the 
margins of the document containing the list of objectives. 
These comments v;ere then brought to and discussed at a 
meeting of the entire department. The consultants then 
collated comments and rewrote unclear objectives, aIso^ 
objectives judged as universally "irrelevant*' v'ero removed 
• from the list. A final "objectives evaluation form" was then 
developed using the revised and shortened list of objectives. 
This form allowed each instructor to rate the relevance of 
each objective and suggest th3 most appropriate method of 
instruction for the objective (see Appendix) . 

Pi scussion 

A surprising amount of redundancy among the objectives 

for different parts of the program was uncovered by this 

method. In addition, for thn first time, adequate written 
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descriptions of what v/as expected of inntructors and students 
in each aspect of the program wore generated. The intent 
is to use the written statements of objectives for a variety 
of purposes within the departmental program. They v;ill be 
useful in designing exams ^ informing students at the begin- 
ning of the course about the objectives^ and also in surveying 
student reactions to the program. The latter could include 
student ratings of the clarity and relevance of objectives 
and the degree to which they were actually implemented and 
achieved. 

Applications of this method promise several benefits. 
The method introduces university professors to the concept 
of objectives in a relatively painless v/ay. It promotes open 
discussion on goals and brings clarity to statements of 
objectives since many instructors examine them. It brings 
the evaluation consultant into close personal contact with 
staff members and the content of the courses. It provides 
a written statement of objectives useful for many other 
purposes* Most of all^ perhaps^ the method sets into motion 
a more complete and formative evaluation that seems to strike 
more deeply into couroe design than evaluations which rely 
on student questionnaires and open departmental discussions 
on "how things are going." 

In addition^ this method promoted the notion that 
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objectives determine the appropriate method of instruction 
rather than the reverse. It appears to bo politically viable 
in that all members of the department have direct Input into 
the process. 

One change suggested by this experience would i^e to 
begin with a short workshop for the group leaders on stating 
objectives. \Vhile several of the leaders had experience in 
this, others didn't. The mere the subject matter experts 
know about the procedures of designing and evaluating instruction 
the moj:e effective and long-lasting the resultr of such a 
procedure are like3-y to be. 

Similarities and differences betv;een this technique and 
that of the Delphi technique developed by the Rand corporation 
(Gordon and Helmer, 1964) and elaborated by a number of in- 
vestigators (e.g., Turoff, 1970; Pfeiffer, 1968), was pointed 
out to the authors after the project was undan^ay. The Delphi 
technique suggests several refinements that are c.ppropriate 
for particular settings. If the department or instructors 
involved have a history of poor communication or internal 
divisivencss , the small groups used in the current method 
may not function well. In this case, the individual question- 
naire approach of Delphi v/ould reduce the negative effects 
of problems in group communication. Delphi questionnaires 
are completed anonymously and tabul<itions do not identify 
individuals or even subgroups of individuals who favor certain 




objectives. Also, Delphi usually includes the documenting of 



minority opinions (by way of brief written arguments) along 
with tabulations of ratings on objectives of a group* This 
information is fed back to each member, v/ho is then asked 
if he wishes to change his rating and to again state his 
reasons for changing or staying with his earlier rating. 
Delphi sometimes results in a strong set of pros and cons 
rather than group consensus. In contrast, the current 
method niay create a false consensus by applying group pressure 
to persons expressing minority opinions in workshops or de- 
partmental meetings. 

The importance of having outside consultants in the 
current technique or in Delphi is dufficult to judge, it 
would seem that only an exceptional person could be both 
departmental member and objective data-gatherer and summarizer. 
Outside help may increase the effectiveness of such a process 
but may also cause some resistance to the implementation of 
such a method in some university settings. 
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Figure 1 

Ordered List of Procedures 

1) DGj^artment chairman (or course director) meets with con- 
sultants to decide if the method is appropriate fur the 
needs of the department or course. Department Chairman 
(or course director) chooses group leaders perhaps using 
guidelines of the consultants^ for groups? representing 
segments of the course or curriculum* 

2) *Cor*sultants conduct workshop for group leaders on stating 

obj ectives. 

3) Consultants meet with each group (n<6) to generate objectives 
for a segment of the course or curriculum. 

4) Group leaders send lists of objectives to Consultants. 

5) Consultants revise list in consulation with qroup lead3rs 
and form composite list of department objectives, which 
are then distributed to each member* 

6) **A11 members of the department meet to discuss composite list. 

7) Consultants prepare objectives-evaluation rating sheets. 
(See Appendix) 

8) Department members evaluate objectives. 

* Not actually used in this case study, but suggested as a 
result of this experience. 

** If many disagreements, strong minority opinions, or evidence 
of group communications problems have appeared by this point, 
a procedure similar to advanced states of the Delphi technique 
may be needed here rather than a large group meeting. Individuals 
would respond independently to the list of objectives. 



Appendix 
Objectives Evaluation 



Form 



Directions: Below you will find the objectives recently 
generated by your department. Will you please rate each 
objective by checking the appropriate category. By rating 
an objective "relevant" or "irrelevant" you would be 
agreeing that the department ^ not necessarily you, should 
or should not teach the objective. If you think that further 
discussion or clarification is necessary, please check 
"uncertain" and use the "comments" space accordingly. 
Further, please chock the appropriate method of instruction 
for each objective. If appropriate method is not listed, 
please describe under Teaching Method. 




ERIC 



